This adds a new merge scheduler implementation that uses a (new)
dedicated thread pool to run the merges. This way the number of
concurrent merges is limited to the number of threads in the pool
(i.e. the number of allocated processors to the ES JVM).

It implements dynamic IO throttling (the same target IO rate for all
merges, roughly, with caveats) that's adjusted based on the number
of currently active (queued + running) merges.
Smaller merges are always preferred to larger ones, irrespective of
the index shard that they're coming from.
The implementation also supports the per-shard "max thread count"
and "max merge count" settings, the later being used today for indexing throttling.
Note that IO throttling, max merge count, and max thread count work similarly,
but not identical, to their siblings in the ConcurrentMergeScheduler.

The per-shard merge statistics are not affected, and the thread-pool statistics should
reflect the merge ones (i.e. the completed thread pool stats reflects the total
number of merges, across shards, per node).

…ep up with the merge load (elastic#125654)

Fixes an issue where indexing throttling kicks in while disk IO is throttling.
Instead disk IO should first unthrottle, and only then, if we still can't keep up with the merging load, start throttling indexing.

Fixes elastic/elasticsearch-benchmarks#2437
Relates elastic#120869

The intent here is to aim for fewer to-do merges enqueued for execution,
and to unthrottle disk IO at a faster rate when the queue grows longer.
Overall this results in less merge disk throttling.

Relates elastic/elasticsearch-benchmarks#2437 elastic#120869

Fixes elastic#125639
Relates elastic#120869

…gMergeTasks (#126058)

Fixes #125842 Relates
#120869

…gMergeTasks (elastic#126058)

Fixes elastic#125842 Relates
elastic#120869

…gMergeTasks (elastic#126058)

Fixes elastic#125842 Relates
elastic#120869

…gMergeTasks (#126058) (#129516)

Fixes #125842
Relates #120869

…gMergeTasks (#126058) (#129515)

Fixes #125842
Relates #120869

This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
#120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.

…9464)

This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
elastic#120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.

…129629)

This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
#120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.

…9464) (#129628)

* Deprecate indices.merge.scheduler.use_thread_pool setting (#129464)

This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
#120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.

* Update warning msg

…9464)

This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
elastic#120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.

…9464)

This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
elastic#120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.

This documents the new threadpool-based merge scheduler, which is disk
space aware, and blocks merges when disk space is low. The code changes
were mostly introduced in
#120869 and
#127613 .

This documents the new threadpool-based merge scheduler, which is disk
space aware, and blocks merges when disk space is low. The code changes
were mostly introduced in
elastic#120869 and
elastic#127613 .

This documents the new threadpool-based merge scheduler, which is disk
space aware, and blocks merges when disk space is low. The code changes
were mostly introduced in
elastic#120869 and
elastic#127613 .

This documents the new threadpool-based merge scheduler, which is disk
space aware, and blocks merges when disk space is low. The code changes
were mostly introduced in
#120869 and
#127613 .

This documents the new threadpool-based merge scheduler, which is disk
space aware, and blocks merges when disk space is low. The code changes
were mostly introduced in
#120869 and
#127613 .

…k space aware, and blocks merges when disk space is low. The code changes were mostly introduced in elastic#120869 and elastic#127613 .

…k space aware, and blocks merges when disk space is low. The code changes were mostly introduced in #120869 and #127613 . (#130530)

This documents the new threadpool-based merge scheduler, which is disk
space aware, and blocks merges when disk space is low. The code changes
were mostly introduced in
elastic#120869 and
elastic#127613 .